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"VIDEO ENCODING METHOD" 

FIELD OF THE INVENTION 

TTie iwesent invention generaHy relates to the Add of data compression 
and, more speciRcaily, to a method of encoding a sequence of ftames, composed of 
picture elements (pixels), by means of a three<Bmenslonal (3D) subband decomposition 
involving a filtering step applied, in the sequence consideied as a 3D volume, to the 
spaOal-temporal data which correspond In said sequence to each one of successive 
groups of frames (GOFs), these GOFS being themselves subdivided into successive pairs 
of frames (POft) including a so^alled previous frame and a socalled current frame, 
said decomposition being applied to said GOfis together with motion estimation and' 
compensation steps perfomied In each GOFon saids POft and on corresponding pairs 
of low-frequency temporal subbands (POSs) obtained at each temporal decomposition 
level. 

The Invention also relates to a computer programme comprising a set of 
instructions for the Imptementatton of said encoding method, when said progranmie is 
earned out by a processor Included in an encoding device. 

BACKGROUND OF THE INVENTCON 

In recent years, three-dimensional (3D) subband analysis, based on a 3D, 
or {2D+t), wavelet decoinposition of a sequence of Irames considered as a 3D volum ' 
has been more and more studied for video compression. The coefficients generated by 
the wavelet transfbmi constitute a hierarchical pyramid in which the spatio-temporal 
relationship is defined thanks to 3D orientation trees evidencing the parent-offeprlng 
dependencies between ooefRdents, and the in-depth scanning of the generated 
coefficients in the hierarchical trees and a progressive bitplane encoding technique lead 
to a desired quality scalability. The practical stage for this approach is to generate 
motion compensated temporal subbands using a simple two taps wavelet filter, as 
Illustrated In Rg.l for a GOF of eight frames. 

In ttie illustrated Implementatioh, the input video sequence is divided into 
Groups of Frames (GOFs), and each GOF, Itself subdivided into successive couples of 
frames (that are as many inputs for a so-called Motton-Compensated Temporal RKering, 
or MOT module), is first motion-compensated (MC) and then temporally filtered (TF) - 
The resulting low fiequency (L) temporal subbands of the fit^t temporal decomposition 
level are further filtered (TF), and the process may stop after an arbitrary number of 
decompositions resulting In one or more low frequency subbands called root temporal 
subbands (in the illustration, an example with two decomposition levels resulting in two 
root subbands LL is presented). In the example of FIQ.l, the frames of the Illustrated 
group are referenced Fl to F8, and the dotted arrows correspond to a high-pass 
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stages of decomposition are shown (L and H = first stage ; LL and LH = second stage). 

At each temnnral Hanimn»^;u i • -c^i. ... . . . 



.... ^ "—'»^/'-'-""uun = secona stage 

At each temporal decomposition level of the illustrated group of 8 frames, a group of 
motfon vector fields is generated (in the present example, MV4 at the first level, MVS at 
the second one). 



-Whel«-Haar-mulaFeselua©n^alysls^s^lsed^b^ti1e^FnporaH 



decomposition, since one motion vector field is generated behveen eveo. two ftames fn 
the considered group of frames at each temporal decomposition level, the number of 
motion vector flelds is equal to half the number of frames in the temporal subband I e 

. "'"^attheflnstlevelofmoaonvectorfleldsandtwoatthesecondone.MoOon 
^imatfon (ME) and motion compensation (Mq are only performed evenr two fi^mes 
of the inputsequence (generally fn the fon«a,d way), due to the temporal down- " 

sampling by two of the Simple wavelet filter, using these very simple filters, each low 
frequency temporal subband (L) represents a temporal average of the input couples of 
frames, whereas the high frequency one (H) contains the r^dual error after the MOF 

Unfortunately, the motion compensated temporal filtering may raise the 
problem of unconnected picture elements (or pixels), which a.^ not filter^f at aH (or 
also the problem of double-connected pixels,.which ara filtered twice). The number of 
urK»nnected pixels represents a weakness of a 3D subband codec approaches because 

■thighV impacts the resulHng picture qualily(paitlcularly.inocdusion regions) itis 
especially true for high motion sequences or for final temporal decomposition levels 
Jhere the temporal corralabon is not good. The number of these unconnected pixeL 
depends on the dense motion vector field that has been generated by the motion 
estimation. 

do not take rnto accountthe number of unconnected pixels that will be the i^ltof 
Tr^"""""""""- ^°^^P'^'^'=^ted algonthms use a rate/dlstomon criterion 
Which tends to minimize a cost function that depends on the displaced difference 

energy(dlstortlon)and the number Of bib spenttotransmlt the motion vector (rate) 

For example, the motion search returns the motion vector that minimises : . . 



(1) 



-th m = («,,.«.,f being the motion vector, ^^u>.,P,f bang the p^iction for the 
motion vector, and A^,^ being the l^grange mulHplier. The rate term i?(in-p) 
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represents the motion information only and SAD fe used as distortion measure. It is 
computed as : 

• SAD{s,c{m))^ YJ[A^fyyc[x-m,,y-mM (2) 

With s befng the original video signal, cbang the coded vfdeo signal and B being the 
blodc size (note that B can be 1), 

Unfortunately, these algorithms do not take into account the distortion 
introduced by unconnected pixels during the inverse motion compensation because usually 
■ these optimizations are applied to hybrid coding for which the inveree moHon 
compensation Is not peribrmed. A previous European patent application n°02293062.2 
(PHFR020136), flted by the appFKant on December 11, 2002, has then proposed a solution, 
for avoiding this drawbadc Said solution, in which the set of unconnected pixels is now 
talcen into account in the distortion measure, relates to a method of enojding a sequence 
of frames, composed of picture elements (pixels), by means of a three-dimensional (3D) 
subband decomposition involving a filtering step applied, in the sequence considered as a 
3D volume, to the spatiaJ-lemporal data which correspond in said sequence to each one of 
the successive GOFs. TTiese GOEs are themselves subdivided into successive paiis of 
frames (POFs) including a so-called previous frame and a scHalied current frame, said 
decomposition being applied to the GOFis together with motion estimation and 
compensation steps peribmied in each GOF on saids POFs and on corresponding pairs of 
low^requency temporal subbands (POSs) obtained at each temporal decomposition level. 
The process of motion compensated temporal filtering leads In the previous frames on the 
one hand to connected pixels, that are filtered along a motion trajectory corresponding to 
motion vectors defined by means of said motion estimation steps, and on the other hand to 
a residual number of so^lled unconnected pixels, that are not filtered at ail. Each motion 
.estimation step then comprises a motion search prwided for returning a motion vector that 
minimizes a cost function depending at least on a distoision criterion Involving a distortfon 
measure, said measure distorsion being also applied to the set of said unconnected pixels. • 

l^ore precisely, for taking into account the set of unconnected pixels in the 
distortion measure it has been proposed to introduce a new rate/distortion criterion that ' 
extends equation taking Into account the unconnected pixels phenomenon. This is 
illustrated in equattons (3) and (4) : 

K{ta) = 7(m) + Xy^com^ ■ D{SummEciw (=«)) (3) 
A-(m) = S40(5,c(m)) + At«vaaMiraaH> ■0(Ww«vaTO(m))+;i«onrw •/?(m-p) (4) 

with Di^mcoHmcmMC^ being the distortion measure for the set Syffa>NmciBD of 
unconnected pixels resullJng from motion vector m . Several distortion measures can be 
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unconnected pixels for the motion vector under study. Nevertheless, the ,^ set of 
un^nnected pixels resuIHng from e motion sean:h can be computed only when the motion 
vector information is avaiiabie for the whole frame, and an optimal solution can ha„lly be 
acnjevaoie. 



-^b^DptlmaHmplenientatlorrharthBfrbBBTrpfo^^ 



and >t is here recalled. For a gTven part of the hnage to be motion compensated (a part of 
«,e, mage can be a pixel, a blodc of pixels , a macmblodc of pixels or any region pn>vlded 
. that the set of parls covers the whole image without any overlapping) and for a given 
. motfon vectorcandldate m, a temporary inverse motion compensation Is applied, theset 
of unconnected pixels Is Identified and i)(^„^(m)) <^n be evaluated. The 
current K{m) value can be computed and compared to the current minimum value 

(m) to check if the candidate mohon vector brings a lower K{m) value When all 
«^«nd«ate have been tested, the (final) Inverse mo«on compensation Is applied to the 
^ -^^t- fldentiiying a,nneC«d and unconnected pixeb). Tl,e next part of the image 
can then be processed, and so on up to a complete processing of the whole Image 

H ,K, K implementatfon, the resulting decisions a,^ not 

spatially homogeneous over the whole image : for the nrst part of «,e image to be motion 
comp««ated, the set Of unconnected pixels Is empty, while the probability of unconnected 

can lead to heterogeneous ^tlal dbtorsions. 

SUMMARY OF THE INVENTION 

IMS therefore an ol^ect of the invention to avoid such a drawback and to 
propose a Video encoding method In whfch the pmblem of heterogeneous fitment 
■ r^u^' "^"^ 'mplemematlon recalled above Is discarded or at least 

To this end, the invention relates to a method of encoding a sequence of 
ft^es, <.mposed Of pIcturBelemente (pixels), by means of a three<limenslonaI (3D) 

D vtirTT""" ' "^'"^ "-'^ered as a 

3D volume, to the ^atial-tempor^i data which correspond In said sequent to each" one of' 
successive groups of flames (60F.), these GOB being themselves subdivided Into 
successh^e pal«. of ftames (POFs) including a so-called previous frame and a so-called 
cunen frame, said decomposltfan being applied to said GOft together with motion 
esbmabon and compensation steps performed In each GOF on salds POB and on 
corresponding pairs of low-frequency temporal subbands (P0&) obtained at each temporal 
decompc^^on level, th. process of mohon compensated tempo.l «tedng leTd^gtre " 
previous frames on the one hand to a,nnected pbcels, that are alo'g a mr^on 
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traJectDfy corresponding to motion vectors defined b/ means of ^id motion estimation 
steps, and on the other hand to a residua} number of so-calJed unconnected pixels, that are 
not filtered at all, each motfon estimatron step comprising a motion search provided for 
returning a motion vector that minimizes a cost ftinction depending at least on a distorsion 
criterion Involving a distortion measure, said measure distorsion being moreover applied to 
the set of said unconnected pbcels. according to the measures and rules defined In claims 2 
and 3. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The present Invention will now be described, by way of example, with reference to the 
accompanying drawing In which : 

- Fig.l shows a temporal multinKolution analysis with motion compensation. 

DETAILED DESCRIPTION OF THE INVENTION 

In order to discard the problem of heterogeneous spatial distortions observed 
with the previous Implementation, it Is now proposed to minimize the global 
criterion ^ lC(m) for all parts of tiie whole Image, which can be done with a multiple- 
pass Implementation including the following steps. 

Rrst^ for all tiie parts of the Image, the optimal motion vector m is 

opt 

computed, as wdl as a ^ of N,^ sub-opHmal mobon vectors. } that provide 

the minimum values fbr J(m) of equatfon (1) (the number of unconnected pixels is not 
used at this stage). The number of sub-optimal vectors N,^^, is implementation 
dependent. For all these vectors, the corresponding value for the criterion J{m) is stored 
so that /(m„p, ) and |/(m„b^ )} are generated. Then an inverse motfon compensation 
. is applied fbr the optimal motion vectors so that ^K{m^ ) can be computed 

at I parts 

(note that not the optimal yahie for TiCfm) because m is 

optimizing J(m) and not ^(m) ). From the list of sub-optimal vectors, the candidate 
motion vector m„^,^.. minimizing |{/{m^ )}- (/(m«^^^ )| is then selected (note 

m^^,^ can be a vector of any part of the current Image). For the srt of optimal 
motion vectors and the candidate vector (in place of the optimal vector for the 
corresponding part of the image), an inverse moBon compensation is applied 
and 's again computed. If its value is lower than TKim J, the optimal 

at t ports ,T^' opt/ r 

all parts 
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fe dfecarded fiiom the fist of sub-optimal vector. TTien a new c3ndidate is 
selerted and the same mechanism Is applied until the list of sub-optimal vector Is emntv 
.nordertDobtalntheoptlmalsetofmotlonvecto^. "'a' vectors Is empty. 
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CLAIMS : 

1. A method of encoding a sequence of frames, composed of picture elements 

(pbcels), by means of a three-dimensional (3D) subband decomposfbon involving a fiitering 
step applied, In the sequence considered as a 3D volume, to the spatial-temporal data 
which correspond In said sequence to each one of successive groups of frames (GOFs), 
these GOI=s being themselves subdivided Into successive pairs of frames (POFs) including a 
so-called previous frame and a so-called current frame, said decomposition being applied 
to said GOFs together with motion estimation and compensation steps perfomied In each 
GOF on saids POFs and on corresponding pains of low^frequency temporal subbands (POSs) 
obtained at each temporal decomposition level, this process of motion compensated 
temporal filtering leading in the previous frames on the one hand to connected pixels, that 
are filtered along a motion trajectory con^pondlng to motion vectors defined by means of 
said motion estimation steps, and on the other hand to a residual number of so-called 
unconnected pixels, that are not filtered at all, each motion estimation step comprising a 
motion search provided for returning a motion vector that minimizes a cost function 
dep'encfing at least on a dlstorsfon criterion involving a distortion measure, said measure 
distorsion being also applied to the set of said unconnected pbcels. 
2. An encoding method according to daim 1, In which said motion search is 

provided for minimizing the following expression (1) : 

Am) = SAD{s,c{m)) + .R{m -p) (i) 



witii m=(m,,»t,f being tiie motion vector, p^(j>,,p^f being tfie prediction for the 
motion vector, .Ji^^nm being tiie Lagrange multiplier, tiie rate term J?(m-p) 

representing the motion information only, and SAD used as distortion measure being 
computed as : 

SADMn^))^ f,\slx,y]-<{x-m^,y^m]\ (2) 

with s being the original video signal, c being the coded video signal and B being the . 
block size, and in which the distorsion criberion extends equation (1), taking Into aomunt 
the unconnected pixels phenomenon for the minimizing operatton that is now applied to 
the following expression (3) : 

^(nj) = 7(m) + AuMZMAiBCTSD • l^^imxMNEam ("»)) (3) 
or f<ira) = SWisMm))+Xummm:m>0(Su,^cm«8:TwMh (4) 
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unconnected dIj»i«: kk» itKn« u „ . 



unconnected pbceb resulting from the moHon vector m 

L distort, 'w" T?' ""^ "^"^ '^^ ^'"5 — t 

thed,sto^ondueto^u„co„„ectedpt.,s,«,eg,oba,.terto„2:taHpar^Kn^^ 

Ca) the optimal motion vector Ss comnnteH ^ 



00^""°" "^'•'"-'^ - we « asa setof N^sub. 

opbmal mctroa vectors that p^vfde the minimum values for 

JCni) ; 

(b) '^a«thesevectors,thacorrespondingvalueforthecrit«rionj(m)l^ 
sto-^/'nordertogenerateJCm^andam^^; 

f^^^"'"^^'^«™«°""«'Pe™atid„.sappHedfbrtheoptimalmot,anvectoR. 
mop. in order to compute ^ [all parts] K(a^ ; 

,/^^'7*^"^°f="b-optlmalvedn«,tf,ecandidatemotfonWm ^ 
fs selected; 

(e) for the set of optimal moUon vecto;. and the candidate vector, an lm««e 

n^ot^n compensa^on is applied, ino^ertocomputeag^ini: can partslKCm'r 

(Oifthe value of ^ [a., par..] K(m) lower than ^ fa« parts, K(m^, 
«,eop«ma..,uec.n^^ 

Cg)nnally,m<^,(sdiscardedfTDmthelistofsub^ptimai vector • 

A computer programme comprising a set of Instructions for the 
implementation of a method according to claim 3 «ih»n e ,w 

P~«-or indudedlnan encoding de'^ ' ""'''"'"""^^^^^ 
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Abstract 

The Invention relates to a method of encoding a sequence of frames by means 
of a three-dimensional (3D) subband decomposition involving a filtering step applied to the 
spatial-temporal data corresponding to successive groups of frames (GOFs), and to a 
recursive implementation of said method. The GOFs are subdivided Into successive pairs of 
frames (POFs), and the decomposition Is applied to said GOFs together with motion 
estimation and compensation steps performed on saids POFs and on corresponding paire of 
low-frequency temporal subbands (POSs) obtained at each temporai decomposition ievei. 
The process of motion compensated temporal filtering leading in the previous frames on 
the one hand to connected pbcefe, that are filtered, and on the other hand to a residual 
number of unconnected pbcels, that are not filtered, each motion esHmatfon step comprises 
a motion search provided for retuming a motion vector that minimizes a cost fuiicOon 
depending at least on a distorsion criterion, said aiterion talcing Into account the 
unconnected pixels phenomenon for the minimizing operation, itseif based on spedftc rules 
allowing to obtain the optimal set of motion vectors. 
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